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OVERVIEW 


ollege students have a better chance of succeeding in school when they receive high-quality 

advising. High-quality advising, when characterized by frequent communications between 

advisers and students, early outreach to students showing signs of academic or nonacademic 
struggles, and personalized guidance that addresses individual student needs, is ideal. It can be a 
crucial factor in student outcomes, academic and otherwise. But strained financial resources and 
personnel constraints at many community colleges and broad-access universities — where advisers 
often have large caseloads — means most students’ advising experiences are limited to a handful of 
interactions that focus mostly on course registration. 


Technology tools may help. Integrated Planning and Advising for Student Success (iPASS) is an 
initiative designed to help colleges redesign advising practices using technology. The iPASS goal is 
to use technology to support reforms aimed at improving communication and outreach to students, 
identifying and supporting struggling students, increasing the number and quality of advising ses- 
sions, and to ultimately improve students’ short- and long-term academic outcomes. 


To study how technology can support advising redesign, MDRC and the Community College Research 
Center partnered with three institutions already implementing iPASS that wanted to enhance their 
existing advising practices, including their standard iPASS services: California State University, Fresno; 
Montgomery County Community College in Pennsylvania; and the University of North Carolina at 
Charlotte. In general, the colleges focused their enhancements on three areas: expanding informational 
messages to students, identifying and supporting students who are struggling, and redesigning advis- 
ing sessions. The study used a randomized controlled trial design, assigning students at random to a 
group eligible to receive the enhanced iPASS services for two semesters, or to a group eligible to receive 
the colleges’ standard services, including standard iPASS. Thus, the study is not a test of iPASS, but 
a test of enhanced advising relative to standard practice under previous iPASS activities. 


This final report from the project, which began in 2016, summarizes the program’s implementation and 
its effects on students’ academic outcomes for four semesters after study entry. The implementation 
research indicates that the enhancements led to a small difference in the student experience — the 
colleges saw an increase in communication with students, a small increase in the number of meet- 
ings between students and advisers, and, at two of the three colleges, an increase in the proportion of 
students who, along with their advisers, received early alerts if the student was struggling in a given 
course. However, the updated impact findings show the enhancements did not have positive effects 
on academic outcomes. Additionally, the enhancements caused a small reduction in credits earned 
at one college, most likely because some students could not register for the next semester until they 
had attended mandatory advising sessions. 


Mounting evidence from the study of education reforms suggests improving students’ academic out- 
comes requires more substantial changes to their college experiences. Reflecting a tension between 
scale and intensity, the study colleges managed to bring their iPASS work to more students, but the 
enhancements were not intense enough to create a substantial difference in students’ experiences. 
Adopting new technology and using it to redesign advising is an iterative process, and it takes time. 
This effort should be viewed as one step in the process of achieving broader change. The findings 
from the project may serve as a useful guide to colleges as they move forward. 
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Introduction 


The labor market has changed dramatically over the past several decades. Technological advances, 
increased international outsourcing, and growing automation increased wages for workers with 
higher education levels and depressed wages for those at the bottom of the earnings distribution." 
As a result, postsecondary education assumes greater importance as a pathway to good jobs and a 
middle class standard of living. 


While college enrollment has increased over time, completion rates have remained low. Only about 
60 percent of students who start at a four-year college earn a degree within six years, and comple- 
tion rates are even lower for low-income students.” Completion rates are also lower at community 
colleges. These colleges serve about 40 percent of all college students, a disproportionate number 
of whom are low-income and underrepresented students, such as Black and Hispanic students and 
older students.* Among first-time students entering public two-year colleges, only about a third 
graduate with a degree or certificate within six years.* 


Low completion rates reflect the many challenges students face, including the cost of attending col- 
lege, balancing work, family obligations and school, low basic academic skill levels, and difficulty 
navigating the college environment. In response, colleges and universities have attempted to address 
these challenges with a variety of strategies, such as increased financial aid, blocked scheduling (in 
which students take many of the same courses together), and other student support.° 


Enhanced advising plays a crucial role in these efforts. Advisers can provide critical support to stu- 
dents as they enter and progress through college. In an ideal world, advisers would communicate 
with students frequently, reach out early to them when they show signs of academic or nonacademic 
struggles, and provide personalized guidance to meet their needs.® 


Unfortunately, most students do not have this kind of advising experience, often because of schools’ 
inadequate financial resources and personnel constraints, particularly at community colleges. Advisers 
at community colleges and broad-access universities generally have large caseloads, in some cases 
as many as 1,000 students per adviser, severely limiting the time available for monitoring and inter- 
acting with students.’ Asa result, advising for many students is limited to a handful of interactions 
during college and focused mostly on transactional tasks, such as registration for courses, rather 
than on addressing challenges they may face. 
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Technology tools may help by increasing advisers’ ability to communicate with students, track their 
progress, and intervene when they are struggling. Many colleges now incorporate some forms of 
technology, such as degree planning tools, into their student support practices.® More institutions 
are also investing in early alert systems that allow faculty to flag students who are missing assign- 
ments or showing other signs of academic difficulties. Technology can support educational reform 
as a foundation for college advisers and staff to change how they interact with and support students, 
ultimately improving the student experience. 


Integrated Planning and Advising for Student Success (iPASS) is an initiative funded by the Bill & 
Melinda Gates Foundation, designed to help colleges invest in and implement technology-based 
advising practices. To date, 45 colleges have received iPASS grants to support technology tools and 
redesigned advising and student support. Some of the colleges’ new tools include communication 
tools, degree planning tools, and early alert systems. 


This report focuses on three iPASS grantees (California State University, Fresno, Montgomery 
County Community College in Pennsylvania, and the University of North Carolina at Charlotte), 
that sought to further strengthen their advising and student support practices and accelerate reforms 
started under iPASS. The enhancements focused on increasing informational communication with 
students, improving the process through which struggling students are identified and supported, 
as well as redesigning advising sessions. Some students were also required to attend one or two 
mandatory advising sessions. 


MDRC and CCRC partnered with the three colleges to assess the implementation and impact of the 
enhancements to iPASS programs. Specifically, the study assessed how enhancements were imple- 
mented at each college, how they changed the student experience, and how they affected students’ 
academic progress in terms of retention and credits earned. The study was conducted as a random- 
ized controlled trial, assigning students at random to a group eligible to receive the enhanced iPASS 
services for two semesters or a group eligible to receive the colleges’ standard services, including 
standard iPASS. Thus, the study is not a test of iPASS, but a test of enhanced advising relative to 
standard practice under previous iPASS grants. 


There have been two reports from the project to date — an early design and implementation report, 
and an interim impact report tracking effects through the program period, the first two semesters 
after students entered the study.° In addition to those publications and this report, there will be 
a forthcoming implementation report documenting the colleges’ experiences using technology to 
redesign advising. The findings indicate that the colleges were largely able to implement their en- 
hancements, although the adoption of new technology is an iterative process that takes time. The 
enhancements led to increased communication with students, an increase in the proportion of stu- 
dents who received early alerts, and small increases in the number of interactions between students 
and advisers. However, the content and nature of the advising sessions did not differ substantially for 
students in the enhanced iPASS group compared with students in the group that received the colleges’ 


8 Tyton Partners and Babson Survey Research Group (2016). 
9 Kalamkarian, et al. (2018); Mayer et al. (2019). 
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standard services. Finally, the enhancements did not positively affect students’ academic progress 
during the first two semesters and led to a small and unexpected reduction in credits earned at one 
college, most likely due to the policy that did not allow students to register for the next semester 
(registration hold) until they attended mandatory advising sessions. 


This report updates the findings from the interim impact report to include semesters three and four 
after students enrolled in the study, or for two semesters after the enhancements were no longer in 
effect. Although there were no observed effects on academic progress during the program period, 
or the first two semesters after students entered the study, it is possible that effects might have 
emerged in later semesters. If students benefit from additional advising early on, for example, they 
may continue to seek out advising in later semesters, potentially leading to positive effects on their 
later academic performance. However, the updated findings indicate that the enhancements did not 
improve student’s students’ school progress in semesters three and four, either in terms of retention 
or total credits earned. In addition, the small reduction in credits earned through semester two at 
one college persisted through semester four. 


The paper also summarizes the implementation of enhancements at each college and concludes with 
lessons on the use of technology to support redesigning advising and student services. The paper 
draws heavily from the two earlier reports and is intended to present an overall summary of the 
project. Although the findings suggest that the enhancements led to small differences in student 
experiences, the project’s results are more broadly applicable as a resource for the field, highlight- 
ing the opportunities and challenges colleges may face in redesigning advising and student support 
using technology. 


The Integrated Planning and Advising for 
Student Success (iPASS) Initiative 


The Bill & Melinda Gates Foundation began funding 19 colleges in 2013 to support iPASS activities. 
Since 2015, the Gates Foundation and other foundations have given 26 additional colleges iPASS 
grants of up to $225,000."° All iPASS grantees also received additional support from EDUCAUSE and 
Achieving the Dream for integrating technology and redesigning advising services over the three- 
year grant period."' Grantees also had access to professional development opportunities through a 
broad network of institutions, technology vendors, and technical assistance partners. Each partici- 
pating college identified individuals across several departments (such as Institutional Research and 
Effectiveness, Information Technology, Student Affairs and Enrollment Management, and Academic 
Affairs) to plan, manage, and implement the integration of technologies and advising reforms. 


10 The 2015 grants were funded by the Bill & Melinda Gates Foundation and The Leona M. and Harry B. 
Helmsley Charitable Trust. 


11 EDUCAUSE is a nonprofit association that supports postsecondary institutions in their use of information 
technology. Achieving the Dream is a nonprofit organization focused on evidence-based efforts to improve 
community college students’ outcomes. 
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The goals of iPASS are improving communication and outreach to students, identifying and sup- 
porting struggling students, and increasing the number and quality of advising sessions through 
technology-based reforms. The ultimate objective is improving students’ short- and long-term academic 
progress. Figure 1 illustrates the theory of change underlying the iPASS model, in which high-quality 
advising helps students clarify their goals and make better academic decisions to achieve those goals. 


The concept of the “ideal advising experience” underlies the motivation for the iPASS program.” 
This ideal experience would begin with the student meeting with an adviser and charting degree and 
career plans. The student would receive regular communications from his or her adviser or other staff 
and information about college support and services. Struggling students would receive targeted and 
coordinated outreach and referrals to additional support, such as tutoring. The advising relationship 
would be sustained and personalized, moving beyond just helping with registration. The iPASS goal 
supports using technology to help transform advising and student experiences closer to this ideal. 


To that end, the colleges implemented a range of technology tools, including early alert systems, 
predictive analytics used to help target struggling students, education planning tools, communication 
tools, and case management software. Examples of how these tools can help include the following: 


e Early alert systems allow faculty to identify students who are missing assignments or struggling 
academically at multiple points during the semester. 


¢ Predictive analytics tools use individual student data, such as demographic information and 
prior performance, to identify students who are likely to struggle academically, assigning them a 
"risk score" based on their predicted likelihood of particular outcomes, such as not persisting to 
the next semester or not graduating. 


Using these tools, advisers can proactively reach out to students at risk of struggling academically 
and offer help. Typically, advisers determine if students need help by interacting with them during 
advising sessions or looking up their grades in a learning management system. Ideally, data provided 
by early alert and predictive analytics systems can help advisers identify at-risk students more ef- 
ficiently, differentiate their interventions based on the nature and degree of individual needs, and 
allocate their limited time to addressing students with the greatest needs. 


e Education planning tools include systems for selecting programs and courses, mapping degree 
plans, and tracking progress toward degree completion. 


These tools allow students and advisers to define clear plans for education and career goals and 
track progress towards those goals. Planning tools can help students map out the specific courses 
required each semester for a selected program. Some tools will also notify students and advisers 
when a student attempts to enroll in a course that is not part of his or her course plan. Information 
gathered using these tools can also guide students’ advising discussions, helping advisers work with 
them to determine individual pathways and stay on track. 


12 Karp and Stacey (2013). 
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RESOURCES 


= Trained student 
support staff 


= Institutional 
leaders who provide 
staff members with 
technology tools 
and professional 
development 


= Technology tools, 
like early alert 
systems, predictive 
analytics, education 
planning tools, and 
communication tools 


FIGURE 1 iPASS Logic Model 


PROGRAM ACTIVITIES 


= Advisers regularly 
communicate with 
students, in-person and 
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= Early in the semester, 
advisers and faculty 
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students 


= Advisers intervene 
with identified students, 
virtually and during 
required advising 
appointments 
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technology tools to refer 
students to support 
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academic and career 
goals 
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career goals 


= Understanding of how 
current courses align 
with long-term goals 


= Support from peers, 
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management and study 
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¢ Communication tools allow electronic messaging and outreach to students, electronic referrals 
to support services, and shared information across staff. 


These tools allow advisers to regularly communicate with students, providing general and personalized 
information, and intervening when necessary. Advisers can use communication tools to electronically 
refer students to resources that align with their needs, such as tutoring or mental health services.'* 


Communication tools can also help advisers and other college personnel coordinate when intervening 
on behalf of a student to ensure he or she gets a coherent message consistent with the entire school’s 
approach. For example, when a student is referred to tutoring, the tutor or other staff member pro- 
viding the service can see the referral, allowing them to follow up with the student if needed. The 
adviser can also see whether and when the student acts on the referral."* 


As another example, a shared note-taking platform allows colleagues to see each other’s notes about 
their interactions with students. When meeting with a student who has been served before, staff 
members can refer to those notes and engage in a more personalized discussion. Staff members can 
align their advising with previously offered information and guidance. 


A study of iPASS implementation at six colleges that received grants in 2013 examined their use of 
technology to transform advising and student services during the first 18 months of their grants. 
Each college put the technology in place, which is an accomplishment in itself, given the challenges 
within any organization to using new data and communication systems.'® Each college made prog- 
ress in transforming student support, although there were challenges. Obstacles included getting 
advisers and other staff to use available data to change how they interact with students and have 
more meaningful discussions with them. Some advisers continued to focus on helping students with 
registration, a more clerical and transactional approach. 


In terms of its impact, earlier research on iPASS indicated that this work to transform student services 
using technology is nonlinear, complex, and often conducted under dynamic internal and external 
conditions. Thus, it is challenging to tease apart the impact of the advising work from the multitude 
of other initiatives taking place at colleges and universities at any given time.'® 


13 Communication tools such as email, learning management systems (such as Blackboard), and case 
management technologies were used to facilitate electronic messaging. 


14 Advisers were typically required to log in to the system to see the referral, rather than receiving this 
information as a push notification. 


15 Karp et al. (2016). 


16 Klempin, Pellegrino, Lopez, and Barnett (2019); Klempin and Pellegrino (2020); Velasco, Hughes, and 
Barnett, (2020). 
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Enhancements to Advising Using iPASS 
at the Three Participating Colleges 


To better understand how technology can support advising redesign, MDRC and CCRC partnered 
with three institutions — California State University, Fresno; Montgomery County Community 
College in Pennsylvania; and the University of North Carolina at Charlotte. These institutions, re- 
ferred to hereafter as Fresno State, MCCC, and UNCC, sought to strengthen their advising processes 
with enhancements that would accelerate and deepen their iPASS reforms. The goal was expanding 
on their existing iPASS work and studying the effects of these enhancements on student outcomes. 


These three colleges were selected for the study showed an interest in making the enhancements and 
had already been identified as strong implementers of technology-based practices under iPASS. The 
team also sought two-year and four-year institutions willing to meet the requirements of the evalua- 
tion design and with adequate sample size. The colleges each received supplementary funding from the 
Bill & Melinda Gates Foundation to offset study participation costs (such as time for research-related 
activities) and to expand the capacity of advising staff to support students. Table 1 presents selected 
features of each college. Further demographic information is listed in Table A.1 in the Appendix. 


The colleges began planning for their selected enhancements in fall 2016. CCRC and MDRC provided 
ongoing support and feedback throughout the planning period, working with college administrators 
and advisers to design the enhancements and providing training on the enhancement components 
and operations to college staff. CCRC and MDRC continued their technical assistance through site 
visits, regular phone calls, and feedback on advising materials through the spring 2018 semester. 


Table 2 presents the components of each college’s enhancement and the differences from standard 
practices. The colleges focused on three areas: expanding informational messages to students, iden- 
tifying and supporting at-risk students, and redesigning advising sessions. 


1. Expanding informational messages 


As part of standard practice, colleges did send some communications to students. At UNCC, for 
example, email and phone messages went to selected students, determined on a case-by-case basis, 
but there was no systematic outreach to students. Enhancements made communication with students 
more frequent, systematic, and sustained. Schools targeted additional outreach to subgroups of 
students with specific circumstances or needs. Email and phone outreach informed students about 
relevant campus resources and activities, including how to get tutoring and deadlines for financial 
aid requirements, as well as more general information, such as tips for preparing for and taking 
exams. The colleges all sent multiple phone or email informational communications each semester. 


For example, at MCCC, students received messages welcoming them to the new semester, remind- 
ing them to complete their MyCareerPlan tool and informational messages about various campus 
resources throughout the semester. At Fresno, students received alert messages if they were struggling 
in certain courses and calls from peer mentors to check in on students who received early alerts. 
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TABLE 1 Insitutional Characteristics 


CHARACTERISTICS FRESNO 

Overview 

Degree of urbanization City: Large 

Level of institution Four-year and 
aAcd . graduate 

Open admission policy No 


Fall enrollment 


Total students 24,405 
Total undergraduates 21,530 
Undergraduate status (%) 
Full-time 87.5 
Part-time 12.5 


Race/ethnicity of undergraduates (%) 


American Indian or Alaska Native, Native Hawaiian or Other Pacific Islander 0.5 
Asian 14.3 
Black or African American 3.0 
Hispanic 49.3 
White 19.9 
Two or more races, Nonresident Alien 8.4 
Race/ethnicity unknown 47 


Financial aid status of undergraduates (%) 


Awarded Pell Grant 57.4 
Full-time first-time students awarded any financial aid 87.4 
Outcomes 
Retention rates* (%) 
Full-time 79.0 
Part-time 47.0 


Completion rate of degree/certificate” (%) 


100% of normal time 15.0 
150% of normal time 52.0 
200% of normal time 60.0 


MCCC 


Suburb: Large 
Two-year 


Yes 


11,480 
11,480 


32.9 
67.1 


0.6 
5.7 
14.3 
6.2 
59.3 
4.7 
9.3 


27.5 
63.7 


65.0 
48.0 


8.0 
20.0 
26.0 


UNCC 


City: Large 


Four-year and 
graduate 
No 


28,721 
23,404 


86.6 
13.4 


0.4 
6.0 
16.4 
9.2 
58.8 
6.6 
2.6 


37.3 
75.9 


82.0 
79.0 


25.0 
55.0 
58.0 


SOURCE: Data from U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary Education Data System 


(IPEDS) 2016-17. 


NOTES: “This represents first- to second-year retention rates of first-time students in fall 2016. 


Completion rates for California State University, Fresno and University of North Carolina at Charlotte are calculated for a four-year bachelor's 
degree (four, six, eight years). Completion rates for Montgomery County Community College are calculated for two-year degrees or certificates. 
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TABLE 2 Summary of Enhanced and Standard Advising Models: 
California State University, Fresno; Montgomery County Community College; 
University of North Carolina at Charlotte 


COMPONENT 


ENHANCED ADVISING 


California State University, Fresno 


Informational communication 


Targeted communication 


Advising sessions 


Training for students on 
MyDegreePlan 


Message welcoming students to semester, 
announcing MyDegreePlan workshops 


Early alert messages at Weeks 5, 7, and 10 
informing students of flags in specific courses 


Call from peer mentor reminding students to 
sign up for MyDegreePlan workshop 


Call from peer mentor checking in on flagged 
students 


Call from peer mentor reminding students to 
sign up for advising appointment 


Call from peer mentor reminding students 
about upcoming advising appointment 


Mandatory advising 


Email campaign sent by adviser to schedule 
appointment 


One-hour session 


Adviser toolbox to guide advising session 


One-hour workshop on MyDegreePlan, with 
opportunity to complete degree plan with 
facilitators 


Montgomery County Community College 


Informal communication 


Message welcoming students to semester, 
requesting completion of self-report survey 


Message instructing students to complete 
MyCareerPlan 


Automated early alert message at Week 3 
informing students of their progress in specific 
courses 


Message requesting completion of second 
self-report survey 


STANDARD ADVISING 


No welcome message required, though 
some advisers may send one 


Early alert message at Week 7 informing 
students of flags in specific courses 


No peer mentor phone calls regarding 
MyDegreePlan 


No peer mentor phone calls to check in on 
flagged students. Email or call from Support 
Net staff for flagged students; process and 
criteria for outreach vary by college 


No peer mentor phone calls reminding 
students to sign up for advising appointment 


Peer mentor phone calls at one of eight 
colleges reminding students about upcoming 
advising appointment 


Advising required only for students who 
are in their freshman year, have reached 75 
credits, or are on academic probation 


Walk-in or scheduled appointments initiated 
by student 


30- to 45-minute advising sessions (varies by 
college) 


No adviser toolbox to guide advising session 


No targeted in-person training on 
MyDegreePlan 


No welcome message required, though 
some advisers may send one 


No message about completing 
MyCareerPlan 


No Week 3 early alert message 


No self-report surveys 


(Continued) 
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COMPONENT 


TABLE 2 (continued) 


ENHANCED ADVISING 


Montgomery County Community College (continued) 


Targeted communication 


Advising sessions 


Message wishing students good luck on finals 


Series of “Did you know?” general 
informational messages throughout the 
semester 


Message responding to early alert surveys 


Message responding to self-report survey 
results 


Message responding to midterm grades 


Mandatory advising for all students during 
their first program semester and a subset 
of students during their second program 
semester 


Advisers send messages to students to 
schedule appointment 


45-minute session 


Adviser toolbox with eight core performance 
areas to guide advising session 


University of North Carolina at Charlotte 


Informal communication 


Targeted communication 


Advising sessions 


Message to welcome students and assign 
“advising homework” or intake form 


Standard messages sent via mail merge 
application and based on enrollment in critical 
progression courses, early alerts, and midterm 
grades 


At least five messages from adviser over the 
course of the semester asking students to 
establish goals and providing feedback based 
on academic performance and early alerts 


Mandatory advising session for students who 
get two or more D or F grades on midterms, 
or one D or F on amidterm in a critical 
progression course 


30-minute session 


Adviser toolbox document to guide advising 
session, with three overarching questions and 
instructions on how to integrate risk information 
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STANDARD ADVISING 


No good luck or end-of-semester message 
required, though some advisers may send 
one 


No “Did you know?” messages 


Practice varies by adviser 


Self-report surveys not administered 


Practice varies by adviser 


Advising not required for students who are 
beyond their first semester at the college and 
not on probation 


Walk-in or scheduled appointments initiated 
by student 


30- to 45-minute advising session (varies by 
appointment reason) 


No adviser toolbox to guide advising session 


No welcome message required, though 
some advisers may send one; no “advising 
homework” required 


No customized messages based on 
students’ risk status 


Communication sent to students on a 
case-by-case basis but with no systematic 
outreach process 


Walk-in or scheduled appointments initiated 
by student 


Length of advising sessions varies by college 


No adviser toolbox to guide advising session 


2. Identifying and supporting students who are struggling 


Each college had an early alert system that allowed faculty to use software to flag academically 
struggling students. However, the alert surveys were rarely issued early in a course, and went out 
after midterm exams or after the seven-week mark. One college lacked a systematic procedure for 
advisers to use to respond to the alerts. 


One iPASS enhancement was collecting these alerts earlier in the semester and, in some cases, at 
multiple points during the semester. Advisers and other support staff responded to these notifica- 
tions with targeted outreach to academically struggling students flagged in the surveys. At Fresno, 
for example, students previously received early alert flags in certain courses only at week seven. The 
enhanced practice sent early flags out after surveys at weeks 5, 7, and 10, charting student progress 
over the fuller course period. 


As part of the effort to identify challenges early, each college also gave a self-analysis survey to 
students in the program group to catalog any self-reported concerns or needs. The plan was that 
advisers and peer mentors would then use survey data to reach out to students if the need could be 
addressed immediately or refer to the survey responses during an advising session that semester. 


3. Redesigning advising 


As part of their standard practices, each college required advising for some students, such as new 
students, those on academic probation, or those declaring majors. All other students outside of these 
designated subgroups typically initiated walk-in or scheduled advising appointments. Sessions were 
typically 35 to 45 minutes long and did not follow a specific format. 


The first enhancement was to expand the required advising appointment to more students. UNCC, 
for example, instituted a mandatory advising session for students in the program group who were 
doing poorly in critical progression courses, defined as receiving a D or F on a midterm exam. At 
Fresno, all students in the study’s program group were required to meet with their adviser before 
registering for courses for the next semester. All three colleges placed registration holds on students 
to compel them to attend the meeting. 


The second enhancement was changing the nature of the advising session interaction. Each college, 
with support from the research team, developed a “toolbox,” a guiding document for advisers to 
use when meeting with students face-to-face.’’ The toolbox contained learning objectives for the 
advising session; probing, open-ended questions to foster student-led discussion; and suggested 
topics for advisers to explore with students, such as non-college obligations, career aspirations, and 
experiential learning opportunities such as study abroad. 


17 The project worked with Elizabeth Wilcox at UC Berkeley to design the toolbox. See https:// 
advisingmatters.berkeley.edu/academic-advising-toolbox. 
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Evaluation 


The evaluation sought to understand how the enhancements were implemented at each college and 
assess their effects on student outcomes. Because the colleges continued making adjustments over 
the course of the study period, the study does not evaluate a static program. As noted earlier, the 
study also does not attempt to evaluate iPASS as a whole at each institution, but the effects of the 
enhancements under iPASS relative to standard iPASS. 


These are the study's key research questions: 


1. How did the colleges enhance their iPASS reforms? 
2. Were the iPASS enhancements implemented as intended? 


3. Did the iPASS enhancements produce a different experience for students, compared with stan- 
dard iPASS? 


4. Did the enhancements produce short- or medium-term gains in student outcomes, compared 
with standard iPASS? 


INTAKE AND RANDOM ASSIGNMENT 


To assess the enhancements’ effects on student outcomes, the study was conducted as a randomized 
controlled trial. Each college first selected a target group for the project, discussed below. Random 
assignment took place across two cohorts, creating the first cohort at the start of the 2017 spring 
semester and the second cohort at the start of the 2017 fall semester. (See Appendix Figure A.1 for a 
timeline of the broader iPASS initiative and the study of enhanced iPASS.) 


Eligible students were randomly assigned to either a program group and offered the two-semester 
enhanced iPASS program, or a control group, which received the institution’s typical advising ser- 
vices via standard iPASS.'® Students in the program group received the iPASS enhancements for 
two semesters. 


PARTICIPATING STUDENTS 


Each college targeted a specific group of students for the project. In general, the target groups con- 
sisted of students at risk of poor performance who were not previously required to visit their advisers 
and in some cases did not already receive targeted outreach from their colleges. 


18 All eligible students were enrolled in the study and could voluntarily opt out of participating at any time. 
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¢ Fresno targeted second-year continuing students who, at the start of the year, had a GPA between 
2.0 and 2.9 and had completed more than 15 but fewer than 75 credits. There are generally no 
policies or institutionalized practices for proactive outreach to these students. Students flagged 
in the early alert system receive emails or calls from support staff, but the process and criteria 
vary by college within the university. Finally, the students targeted for the study typically only 
engage with an adviser if they seek it out. 


¢ MCCC first selected a subgroup of continuing students in degree programs who were not already 
required to meet with an adviser. They were not on academic probation or in selected subgroups 
with required advising services. Within this group, the project was further targeted to students 
who were determined, based on a predictive data model, to have a “low” or “moderate” risk of 
persisting to the next semester. 


¢ UNCC targeted two at-risk groups of students. The first cohort included continuing students with 
a GPA of 2.0 or higher but with a low probability of graduating, based on a predictive data model. 
As discussed in an earlier report, an error in the predictive model meant many students received 
incorrect probabilities and were included in the study erroneously.'® The second cohort included 
transfer students, who tend to drop out at higher rates than continuing students.”° 


Table 3 presents selected characteristics of the students at each college. Women comprise a slight 
majority of the student body at two of the colleges. Their racial and ethnic compositions, with a 
high fraction of Hispanic students at Fresno, for example, reflect regional differences as well as the 
composition of the broader student bodies at the colleges. Most students were traditional college- 
aged at study entry, between ages 19 and 24, although a sizeable share of students at MCCC were 
over age 25, which is typical among community colleges. Finally, all students entered the study with 
some credits earned. Students at Fresno had earned the most credits, in part because that college 
targeted second-year students for the study. 


KEY MEASURES OF SERVICE CONTRAST AND 
ACADEMIC PROGRESS 


The studies collected qualitative and quantitative data to understand how the changes were being 
implemented and to assess their effects on student outcomes. These data were also used to provide 
feedback to the institutions, and the qualitative data informed adjustments to the program over the 
course of the study period. The study tracked implementation of the iPASS enhancements using a 
variety of sources, including site visits to each institution and iPASS program information, such as 
early alert and advising appointment data. Key outcomes used to measure service contrast — the 
difference between the services received between the program and control groups — are early alert 


19 The error in the model has implications for the impact of the enhancements on student progress, since 
some proportion of the study sample was not at risk and thus may not have needed the enhanced services 
to succeed. 


20 Duggan and Pickering (2008). 


Findings and Lessons from Three Colleges’ Efforts to Build on the iPASS Initiative 


13 


SSS —————————— | 
TABLE 3 Demographics of Study Sample, by School 


SCHOOL FRESNO MCCC UNCC 
Female (%) 61.7 59.2 43.4 
Race/ethnicity (%) 
Hispanic 62.5 74 9.5 
White 12.2 56.2 60.1 
Black 2.8 18.0 14.4 
Other 22.6 18.4 15.9 
Age (%) 
18 and under 10.5 4.5 28.9 
19-24 89.5 54.9 66.5 
25 and over 0.0 40.7 4.5 
College credits earned at study enrollment 44.35 37.97 29.71 
Sample size 1,219 2,989 3,803 


SOURCE: MDRC calculations using data from California State University, Fresno; Montgomery County 
Community College; and University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Sample sizes may vary because of missing values. 


notices and number of advising sessions, obtained from iPASS program data. Academic outcomes 
include enrollment in each semester and credits earned over the full period, both of which were 
obtained from the colleges’ student transcript data. 


Implementation 


Across two rounds of qualitative fieldwork the research team uncovered key successes and challenges 
associated with implementing enhancements to advising.*' The research team took an iterative ap- 
proach, working closely with the colleges to qualitatively gauge how the tools and tactics were working 


21 During 2017, the research team conducted two visits at each institution in the spring and fall semesters 
(six total) to better understand how advising redesign, particularly as specified by the enhancements, 
was being implemented. During that time, the study team conducted 238 in-person interviews. In spring 
2018 (the second intervention semester for cohort two students), the team conducted an additional 
18 interviews by phone. During each round of data collection, the team interviewed a wide range of 
stakeholders including advisers, administrators, key project staff members (who oversaw the work), 
and other support personnel such as peer mentors at Fresno and the retention specialist at MCCC. For 
additional information about the implementation research methodology see Mayer et al. (2019). 
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and making adjustments as needed. The research and institutionally based teams communicated 
regularly in meetings and with on-site visits, weekly calls, and ongoing correspondence. The many 
lessons learned during the course of the study may be useful to institutions embarking on advising 
reform work, particularly in the areas of jobs and roles of college staff, student engagement, early 
alerts and student support, enhancements to advising sessions, technology, and perceptions of ad- 
vising and student support at the institutions. 


This report presents a few of the most important indications from the fieldwork, concentrating on 
those the research team judged to be most likely to lend insights into the findings in this report. A 
comprehensive presentation of qualitative findings can be found in the forthcoming implementation 
report (Santikian-Kalamkarian, Pellegrino, & Lopez, 2020). 


e Engaging students in the study’s messaging components was challenging. Strategies worked 
differently at different schools and different contexts. Personalized communications were 
perceived to be most effective. 


At least once during the study time frame, each of the three colleges sent out a student survey that 
encouraged students to report their concerns and any barriers they faced. Advisers overwhelmingly 
reported receiving very low response rates to these surveys (fewer than 10 students per adviser in 
most cases). Advisers said they discussed the survey results during sessions with the few students 
who responded. In some instances, advisers said they were able to proactively reach out to students 
and offer support. One adviser referred to the survey results as a “jumping-off point” for the advis- 
ing session with a student. This small number of experiences indicated that self-reported student 
data may indeed be helpful, though it proved difficult to obtain without an incentive for students 
to respond. In that case, the additional cost of increasing response rates would need to be weighed 
against the benefits from obtaining the data. 


At all three schools, advisers used informational messaging strategies to students through the course 
of each semester. Approximately every two weeks, advisers emailed students with information about 
things like campus resources and upcoming deadlines. The messages also encouraged students to 
see their advisers. When asked about these messages, however, students often could not recall re- 
ceiving them and indicated they tended to read emails that they received from people they know or 
perceived as being personalized and not “mass emails.” 


Despite low student response rates, some advisers indicated they felt the ongoing communication 
strategy were an important part of their efforts because they provided useful information regularly. 
They believed this strategy aligned with their personal commitments to providing developmental 
student support. Some advisers sent students words of encouragement, such as well wishes for final 
exams and study tips. Still, there was consensus among advisers that the messages may have been 
more meaningful to students if they were able to send more personalized messages, acknowledging 
that students tended not to respond to emails from people they didn’t know. Most students did not 
recall receiving the messages from advisers and said they primarily focused on emails from faculty 
and classmates. Notably, emails administered by UNCC encouraging students to make appoint- 
ments with their adviser and including a link to schedule the appointments — dubbed appointment 
campaigns — received strong student response, according to advisers. 
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e Advising sessions were not notably different for students in the program group from those in 
the control group, and there was an uneven engagement by advisers with the toolbox. 


College leaders and advisers at each school reported varying levels of engagement with the advising 
toolbox. Advisers indicated they intentionally did not differentiate their advising approaches toward 
program and control group students because they did not want to treat their students differently. 
Most advisers were generally using a portion of the toolbox, often citing a question or two that they 
found particularly helpful when talking with students. However, few said they regularly referred to 
the document. Many advisers reported recalling a question or two they used during advising ses- 
sions, drawing from memory rather than engaging with the toolbox document itself. Early alerts were 
perceived as helpful to advisers in identifying students who may be struggling, however, response 
to these resources was mixed. 


As Figure 2 illustrates, some colleges used early alerts more than others. As expected, on average 
they were used more with program group students than control group students. While early alerts 
were most often deployed when students were struggling, in some settings they included “kudos” 
to praise students who did well. At UNCC, where faculty could use the early alert system to provide 
positive feedback during one semester of the study, students said they appreciated those messages 
and felt assured they were on the right track. 


Some implementation challenges had the potential to diminish the efficacy of early alerts. One is- 
sue was that if faculty triggered an alert but did not specify why — attendance, grades, test scores 
— advisers could not reach out to students with targeted and personalized support to the extent 
they would have liked. Moreover, many advisers indicated that when they had many students with 
early alert flags, they only had the time and capacity to reach out to the students with multiple flags 
or those deemed most in need. Some advisers expressed concern about the tone and content of the 
messages. One adviser referred to early alerts as potentially “scary” to student recipients. 


Predictive analytics data were also available to advisers at some colleges, which helped them better 
target conversations with students during advising sessions. However, these data were sometimes 
flawed or out of date, and college leaders and advisers sometimes mistrusted them. At UNCC and 
MCCC, advisers reported not using the predictive analytics risk scores for communication with 
students — virtually or in person — because they did not believe them accurate or actionable. At 
UNCC, the predictive analytics algorithm mistakenly identified some strong performing students 
as “at risk.” The predictive analytics tool at MCCC coded students with demographic variables that 
ostensibly put students at risk — such as their residential zip codes — but advisers could not use 
those data in a meaningful way to assist students. 


Students had mixed responses to early alerts. Most students believed early alerts were a good thing 
for those who were struggling, and some students said they especially appreciated the “kudos.” Some 
students said they felt motivated and developed a sense of urgency to improve their academic per- 
formance after getting an alert. However, other students said that early alerts were discouraging or 
confusing, particularly if they were not told why they received the alert or did not know the sender.” 


22 Some early alert systems include features that allow the instructor to appear as the sender, while others do not. 


16 | Using Technology to Redesign College Advising and Student Support 


FIGURE 2 Early Alerts by Semester and Number of Alerts in the First Year 
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FIGURE 2 (continued) 


SOURCE: MDRC calculations using data from California State University, Fresno; Montgomery County Community 
College; and University of North Carolina at Charlotte. 


¢ Technology tools helped advisers but were also limited in some ways. 


Advisers at every school said technology helped with degree planning, career planning discussions 
(particularly at MCCC, where advisers used a career assessment tool with their students), scheduling 
advising appointments, and maintaining well-organized documentation for their caseloads. Advisers 
appreciated being able to pull reports and develop lists of students who might need extra support so 
that they could keep an eye on their progress and make them a priority for intervention if needed. 


Students seemed comfortable with scheduling appointments and with using college-based tools, 
including email. Students and advisers at Fresno said students wanted initial face-to-face assistance 
from a helpful person to feel comfortable selecting courses and planning their future semesters. 
Many advisers said much of their initial advising appointments with new students were dedicated 
to helping them feel feeling confident using degree planning tools. 


¢ Colleges adjusted staff members’ job responsibilities and mobilized non-advising staff mem- 
bers as they implemented the program enhancements. However, this required training new 
personnel and adapting business processes to fit new job responsibilities. 


At each of the three colleges, formal and informal leaders in advising departments or operations 
had to oversee the development and implementation of the new advising enhancement procedures, 
as well as performing their existing job responsibilities. In their daily roles, they generated and sup- 
plied lists of students for advisers to contact, provided email templates for messages, and supported 
technology-related inquiries. Overwhelmingly, advisers said they appreciated this leadership and 
felt assured that a person or people “owned” the work and could serve as a source of support. 


Entirely new staff roles emerged during the study as Fresno and MCCC addressed issues with their 
advising capacity. Fresno hired two student peer mentors (a junior and a senior) to conduct informa- 
tional outreach to students receiving enhancements. The peer mentors were later trained to follow up 
with students who had received early alerts. MCCC hired a retention specialist to support advisers 
by conducting outreach to students who were struggling. 


° The study showed evidence of emerging shifts in attitudes at the colleges about how to do 
advising well. This may carry forward the work of advising reform beyond the grant period. 


Engaging in advising redesign and participating in the RCT using an iterative approach facilitated 
institutional learning and growth at all three schools. Several stakeholders reported a shift in atti- 
tudes about advising and student support, indicating the enhancements offered valuable lessons. For 
example, advisers — even those who said the enhancements had made their workloads challenging 


18 | Using Technology to Redesign College Advising and Student Support 


— appreciated the opportunity to participate in the study. They said it brought greater attention to 
the value associated with advising and student support in general. Several advisers said they hoped 
that institutional leaders would direct more resources to advising and student support. Two advisers 
at MCCC indicated that participating in the study helped them develop stronger, more meaningful 
relationships with students. 


Institutional leaders responded positively to this work as well. At Fresno, university leadership 
directed considerable resources toward advising during and following the study. The university 
created two subcommittees consisting of full-time advisers and faculty advisers — one to consider 
ways to assess advising across campus and a second examining the training and development needs 
of full-time advisers and faculty advisers. The university also allocated a large state grant toward 
expanding the peer mentor program, so that they would be able to work alongside students and help 
them learn to use the degree planning tool. At UNCC, university leaders developed and implemented 
a career ladder, with “tiers,” so that advisers could have increased career mobility at the institution. 


Service Contrast 


This section assesses the service contrast created by the enhancements, or the differences in experi- 
ences and services between students in the program and control groups. Expanded communication 
was a notable aspect of the iPASS enhancements. That was assessed through the implementation data 
collected on adviser and student experiences with the student survey and informational messages. 
The other two program components — advising sessions and early alerts — were assessed using 
program data. For these two components, the following discussion focuses on service contrasts in 
each semester between program and control group students. 


To better understand the baseline experience to which the program group is compared, it is important 
to note that the control group students were experiencing “standard” iPASS, which looked different 
at each of the three schools. During the time of this study, the standard iPASS initiative does not ap- 
pear to have led to big changes in students’ experiences relative to the pre-iPASS period. The colleges 
were not implementing iPASS practices intensively with any particular group of students. Students 
could access iPASS services and tools on their own, and while some tools, such as MCCC’s degree 
planning tool, were marketed to students, their use was not consistently encouraged as standard 
practice. As shown below, for example, most students in the control groups did not receive early 
alerts during the follow-up period. 


e Students in the program group were sent more adviser communications than students in the 
control group. 


As mentioned earlier, communications with program group students included a survey administered 
to the students. This helped collect information on challenges students faced or expected to face 
during the semester. Advisers also used mass email tools to send large groups of students informa- 
tion and resources. Although response rates to both these efforts were low, advisers reported that 
the few survey responses they received were useful. 
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e Students in the program group were more likely than students in the control group to meet 
with an adviser at least once. The differences in the average number of advising visits, however, 
were small. 


Figure 3 presents differences in percentages of study participants who met with an adviser during 
each semester after study enrollment. At Fresno, almost all program group students met with their 
advisers in the first two semesters, when the iPASS enhancements were in place, compared with 
about half of the control group students.?* At MCCC, program group students were also much more 
likely to meet with their advisers than control group students. Fewer than 60 percent of control group 
students had adviser meetings in the first two semesters. Because advising appointments are required 
by some colleges at UNCC, the school had a larger proportion of control group students meeting with 
advisers than the other two schools starting in the first semester of the program. Program group 
students at UNCC were more likely to meet with advisers than control group students, although the 
difference was less stark than at the other schools. 


The right panel of the figure presents impacts on the number of times students met with an adviser 
during the first year of study. While the differences in ever meeting with an adviser are relatively 
large in Fresno and MCCC, the iPASS enhancements induced students to have, on average, just one 
or two more meetings with their advisers than they otherwise would have at Fresno and MCCC, with 
even smaller differences at UNCC, as shown in Appendix Table A.2. Thus, the additional advising 
interactions created by the enhancements was small. 


Data on the duration of adviser meetings were not consistently available across the data systems, 
so time spent with advisers could not be quantified accurately in this analysis. In theory, longer 
appointment times and the toolbox used to structure meeting times with program group students 
could have led to better quality advising for students in the program group. However, as reported 
in the earlier implementation section, some advisers adapted some practices they liked from the 
toolbox for use with their control group students, while others did not typically access the toolbox 
when meeting with program group students — so it is unlikely that the advising sessions for program 
or control group students differed much in their content. 


¢ In two of the three colleges, program group students were more likely than control group 
students to receive at least one early alert during the two program semesters. But, on average, 
program group students only received one to two more alerts than control group students 
during the program. 


Figure 2 presents differences in percentages of study participants who received early alerts in each 
semester. The patterns are similar to the adviser meeting rates — many more program than control 
group students in Fresno and MCCC received early alerts in the first two semesters. MCCC appears 
to have made the most use of the early alerts, sending alerts to almost 90 percent of program group 
students during the first program semester, and about 50 percent of students during the second 
semester (which accounts for about 90 percent of enrolled second semester students). As a result, 


23 In fall 2018, Fresno began implementing new technology tools for early alerts, case management, and 
analytics. For the remainder of the data collection time period, faculty and staff could issue early alerts; 
however, the university did not formally solicit the information. 
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FIGURE 3 Advising Contacts by Semester and Number of Contacts in the First Year 
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FIGURE 3 (continued) 


SOURCE: MDRC calculations using data from California State University, Fresno; Montgomery County Community 
College; and University of North Carolina at Charlotte. 


program group students averaged two to three more alerts than control group students over two 
semesters of the study, while at Fresno, program group students received one more alert than control 
group students did, on average. At UNCC, students in both groups received alerts at similar rates. 
The majority of Fresno and UNCC students did not receive any early alerts. While receiving early 
alerts is an important element of adviser-student engagement, students who received better advising 
may also be less likely to be flagged as struggling in the early alerts system. 


In the third and fourth semesters, after the enhancements were no longer offered, differences in 
advising meetings and alerts were no longer apparent between the program and control groups. 


Impact on Academic Outcomes 


The iPASS theory of change shown in Figure 1 is described in the earlier iPASS report.*4 That report 
considers that if iPASS successfully increased adviser-student interaction and facilitated better sup- 
port for students to set and accomplish their academic goals, then short-term academic outcomes, 
such as enrollment and credits earned, would also improve. This section examines whether the 
enhancements implemented by the colleges led to improvements in students’ academic progress. 


Table 4 presents effects on enrollment and credits earned during the two program semesters, as 
well as the two semesters after the enhanced iPASS intervention ended.”° Results showed that iPASS 
enhancements did not increase enrollment in subsequent semesters or credits earned at the colleges. 
As described in the previous section, enhanced iPASS produced small changes in student advising 
experiences, but colleges may not have had the capacity or resources to implement the program 
enhancements at scale. 


It is possible that two semesters of an enhanced iPASS intervention did not allow sufficient time for 
refining iPASS practices in a way that could substantially improve most students’ advising experi- 
ence (for example, revising the early alert system to provide students with more detailed and relevant 
information), which would be necessary for the program to increase enrollment or credits earned. 
Small differences in service contrasts described in the previous sections may also explain the lack 
of impacts on academic outcomes. 


24 Mayer et al. (2019). 


25 Appendix Table A.4 shows some more academic outcomes by program group not covered in the main 
body of the report, including degree outcomes. 
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a ee ee a] 
TABLE 4 Differences in Enrollment and Credits Earned 


STANDARD 
INSTITUTIONS PROGRAM CONTROL DIFFERENCE ERROR ~ P-VALUE 
California State University, Fresno 
Enrollment (%) 
1st semester (with enhancements) 98.5 98.9 -0.4 0.3 0.165 
2nd semester (with enhancements) 94.2 93.5 0.8 1.3 0.565 
3rd semester (after enhancements) 88.2 86.2 1.9 1.9 0.308 
4th semester (after enhancements) 83.3 80.4 2.9 2.2 0.189 
Total credits earned 43.84 42.94 0.89 0.96 0.353 
Sample size 610 609 
Montgomery County Community College 
Enrollment (%) 
1st semester (with enhancements) 97.1 97.6 -0.6 0.4 0.171 
2nd semester (with enhancements) 52.2 55.8 -3.5 1.8 0.050 
3rd semester (after enhancements) 39.6 43.5 -3.9 1.8 0.029 
4th semester (after enhancements) 29.0 30.1 =1A 1.7 0.490 
Total credits earned 12.54 13.37 -0.83 0.43 0.051 
Sample size 1,248 1,741 
University of North Carolina, Charlotte 
Enrollment (%) 
1st semester (with enhancements) 98.2 98.3 -0.1 0.3 0.686 
2nd semester (with enhancements) 89.3 88.4 0.9 1.0 0.354 
3rd semester (after enhancements) 82.9 83.1 -0.2 1.2 0.870 
4th semester (after enhancements) 77.6 777 -0.1 1.3 0.931 
Total credits earned 46.26 46.24 0.01 0.61 0.983 


Sample size 1,902 1,901 


SOURCE: MDRC calculations using transcript data from California State University, Fresno, Montgomery County Community 
College, and University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with 
zero true effect. A lower p-value suggests a higher likelihood of a nonzero effect. 

Estimates are adjusted by pre-random assignment characteristics of sample members. 

Estimates are weighted to account for differences in random assignment ratios across advisers. 

Spring and summer records are combined. Similarly, winter and fall records are combined. 

Transcript records for summer 2019 are not included in the fourth semester estimates. Few students enrolled in any classes 
during the summer terms. 
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On average, about 80 percent of students in the study sample were still enrolled in the fourth semester 
of the study at Fresno and UNCC, while only about 30 percent were enrolled in the fourth semester at 
MCCC. Recall that MCCC is a community college, and Fresno and UNCC are four-year universities. 
The sharp drop in enrollment between semesters at MCCC is stark compared with the two universi- 
ties in the study, but is typical for community colleges.7° The differences in credits earned between 
the schools also reflect differences in retention rates and may also reflect the higher proportion of 
part-time students at MCCC relative to the other colleges; at Fresno and UNCC, students earned, on 
average, about 45 credits over the four semesters of the study. At MCCC, students averaged 13 credits. 
Included in this average are 0 credits for each semester a student was not enrolled in the school. 
Because a much larger percentage of students initially enrolled at MCCC were no longer enrolled 
in later semesters, and because community college students are generally pursuing associate rather 
than four-year degrees, the average credits earned at MCCC was lower than at the other schools. 


During the second and third semesters, the iPASS enhancements appear to have reduced enrollment 
at MCCC by 4 percentage points. The interim impact report found that the reduction in credits at- 
tempted at MCCC in the first semester of the program was driven by the registration hold placed on 
program group students, which prevented them from registering for some classes if they had not yet 
met with their advisers.”’ For the spring 2017 cohort, the hold may have prevented some students from 
enrolling in the summer term. The registration hold on the program group was eventually eliminated 
in spring 2018, which was the second program semester for the second cohort. Nevertheless, the 
reduction in enrollment from these initial holds appears to have persisted beyond the two program 
semesters with iPASS enhancements. 


MCCC program group students earned, on average, 0.8 fewer credits over 4 semesters than control 
group students. Since the ability to earn credits is dependent on class enrollment, the cumulative 
reduction in credits earned can likely be attributed to the negative effects on enrollment in the earlier 
semesters. The earlier iPASS report provides a more detailed description of the unintended nega- 
tive enrollment effect of the registration hold, and a more comprehensive table examining courses 
potentially affected by the registration hold is presented in Appendix Table A.3.7° 


Although not shown in the table, if the results were pooled across all three institutions, the iPASS 
enhancements would show a near zero effect on academic outcomes. 


26 The study participant retention rate, however, is still about 20 percentage points lower than the national 
average at two-year public institutions. The 2019 annual report from the National Student Clearinghouse 
showed that about 54 percent of students who started school in two-year public institutions in 2017 
returned to the same school in 2018, compared with 71 percent at four-year institutions. Since students 
who enrolled in a community college may not take classes in consecutive semesters or may transfer 
between institutions or to a four-year school, the large drop in enrollment in the second program semester 
at MCCC, compared with smaller enrollment reductions at the other two colleges, is not unexpected. 


27 Mayer et al. (2019). 
28 Mayer et al. (2019). 
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Lessons 


This project studied three colleges’ efforts to enhance their existing iPASS work and measure the 
effects of the enhancements on their students. The enhancements implemented did not positively 
impact students’ academic outcomes. Nonetheless, the findings from the study may provide lessons 
to the large and growing number of colleges working to integrate technology into their student sup- 
port efforts. What follows presents some lessons from the project and some more general lessons 
about enhancing student support with the help of technology. 


¢ More intensive interventions and more frequent advising contacts are probably needed to 
impact student outcomes. 


The three participating colleges were all involved in the preexisting iPASS initiative prior to the 
random assignment study, as shown in the timeline in Appendix Figure A.1. For the study, students 
in the program group experienced a mix of enhancements to what colleges had put in place using 
previous iPASS grants and new components. 


The main program components — and, therefore, points of contrast — were additional informational 
communication, additional early alerts and the use of other methods to support identification of and 
intervention with struggling students, the addition of an advising appointment requirement (enforced 
using a registration hold) for a subset of students, and the use of the advising “toolbox” during the 
required appointment. However, these components resulted in an experience for students in the 
program group that did not differ substantially from business as usual. For example, as mentioned 
earlier, although the program increased the number of advising meetings at all three colleges, on 
average, program students at all three colleges only experienced zero to two more advising meetings. 


Though iPASS was not designed to be a comprehensive student support program, given the body 
of research on what works, this level of program intensity — and therefore level of contrast with 
business-as-usual — is likely not enough to improve long-term outcomes.”° Studies of the most 
effective programs showed students meet frequently with support providers. When students meet 
with an adviser or coach only once or twice in a semester, the effects are usually not long-lasting.*° 


In a study of the City University of New York’s Accelerated Study in Associate Programs (ASAP) 
program, which nearly doubled graduation rates, students in the program group met with advisers 
38 times, on average, in their first year in the program, roughly 32 more times than students in the 
control group.*" A replication of ASAP at three Ohio colleges, which led to large increases in gradu- 
ation rates, required students to meet with an adviser twice a month in their first semester — over 
60 percent of students met with their adviser six or more times during that period. Some students 


29 Scrivener et al. (2015); Bertrand, Hallberg, Hofmeister, Morgan, and Shirey (2019); Evans, Perry, Kearney, 
and Sullivan (2018). 


30 Scrivener and Coghlan (2011). 
31 Scrivener et al. (2015). 
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did not have advising requirements in subsequent semesters, but students continued to meet with 
advisers, though less frequently.*? 


In a study of the Detroit Promise Path (DPP) — a less intensive program than ASAP — students in 
the program group were instructed to meet with their coach twice per month. In all four semesters, 
most enrolled students met with a coach six or more times each semester. Students were given a 
financial incentive for meeting with their coaches. The program is still being studied, and early 
findings are promising, though effects appear to be decreasing in magnitude.** 


The Opening Doors demonstration in Ohio offered a program with a less intensive advising require- 
ment: students were expected to meet with an adviser at least twice per semester and received a $150 
stipend after their second meeting. The program generated modest positive effects on registration 
rates but did not affect long-term outcomes.** 


In general, programs that led to smaller increases in advising have not generated large, if any, im- 
provements in academic outcomes. However, increasing advising interactions was not the sole goal 
of iPASS. The iPASS theory of change hypothesizes that technology tools will support more inten- 
sive, personalized advising experiences, even with limited human resources. That is, even though 
students meet less frequently compared with programs like ASAP and DPP, technology could allow 
advisers to use resources in a more targeted way and more frequently than business as usual. This 
would in turn lead to improved academic outcomes. Ideally, technology like early alerts and predic- 
tive analytics would provide advisers with timely information that a student is struggling, allowing 
advisers to focus their limited time on the students who need the most help. With that information, 
the adviser could then intervene with that student digitally or in person to help get them back on 
track. Based on the findings from this study, technology use is not yet supporting this goal to the 
degree required to improve student outcomes. 


Though the programs mentioned above are the most effective, they require time, people and money, 
which can make them challenging to implement at a larger scale. More work is needed to learn lower- 
cost ways to achieve positive results. 


¢ Consider alternative approaches to collecting information from faculty. 


One iPASS goal is for faculty and student support staff to identify students when they show early 
signs of academic or nonacademic struggle, and have support staff intervene in a timely manner. 
Early alert technology supports this goal, in theory. Faculty must complete early alert surveys for 
the early alert system to be successful. At participating colleges, faculty responsiveness to early alert 
surveys varied depending on individual college culture. For example, one college had been using 
early alerts for a longer time, which may explain their higher response rates. At another college, a 
slight improvement in early alert response rates may be attributed to promotional activities college 


32 Miller, Headlam, Manno, and Cullinan (2020). 
33 Ratledge, O'Donoghue, Cullinan, Camo-Biogradlija (2019). 
34 Scrivener and Weiss (2009). 
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staff conducted with the faculty prior to the study to encourage their use. Involving faculty more in 
discussions about data may help buy-in and response rates will likely improve over time, as faculty 
become more familiar with this kind of reporting. 


¢ Registration holds can have unintended consequences, particularly for community college 
students. Consider alternative methods to encourage students to meet with an adviser. 


At MCCC, the sole community college in the study, the program had negative effects on enrollment 
and credits earned, which were likely driven by the registration holds. It is possible that the registra- 
tion hold caused more students to meet with an adviser at UNCC and Fresno State. But at MCCC, 
advisers reported that classes rarely fill up early. Thus, students may have been less motivated to come 
in early — that is, before the semester’s end — for an advising appointment. The additional barrier 
to registration created by the registration hold may have discouraged students from registering for 
additional classes or reenrolling. 


Some effective community college student support programs offer students incentives to meet with 
advisers or coaches.*° These incentives might come in the form of financial incentives (for example, 
transportation vouchers, gift cards, campus bookstore vouchers) or priority registration. The find- 
ings from this study suggest that incentives may be a worthwhile alternative approach, particularly 
for community college students. At a minimum, community colleges should take care as to how, 
why, and when they place registration holds.*° 


¢ Consider alternative approaches to informational communication. 


In addition to receiving personalized messages about their academic status based on early alert data, 
students in the program also received messages with general information about tutoring hours and 
child care availability, among other topics. When interviewed, most students did not remember 
receiving these messages. Most colleges send messages like these to all students, informing them of 
registration dates and services offered on campus. It can be challenging to get students to read emails. 
To ensure students receive this more general information, it may be worthwhile for colleges to try 
to frame this content in a more personalized or action-oriented way, disseminate this information 
using multiple modes of communication, and consider how the message’s sender might influence 
whether a student will open a message. 


Some other strategies that can be used to engage message recipients and prompt them to act are 
rooted in the principles of behavioral economics. Examples include simplifying message content, 
using language that prompts actions, setting deadlines for action, making actions easier to com- 
plete (such as by providing hyperlinks), personalizing content, and using frequent reminders.*” The 
Encouraging Additional Summer Enrollment (EASE) project used behavioral messaging strategies 
to encourage students to enroll in summer courses. Messages were personalized, came from multiple 


35 Ratledge (2017); Scrivener et al. (2015). 
36 For more on this finding, see Mayer et al., (2019). 
37 Richburg-Hayes, Anzelone, and Dechausay (2017). 
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senders, and utilized different modes of communication (for example, email and postal mail).°* The 
program increased summer enrollment by 5 percentage points.*° 


The mode of communication is important. A study of adviser communications found a positive 
relationship between the number of in-person advising meetings attended and an adviser’s use of 
instant messaging and text messaging. However, there was a negative relationship between meet- 
ings attended and the use of Facebook.*® The EASE project found that while some information was 
communicated over both email and mailed letters, many students recalled their mailed letters.” 
These varying preferences make it important to diversify modes to ensure all students are reached. 


It is also important to consider the message’s sender: A survey of 315 students found that students 
were very likely to read emails coming from faculty but less so when coming from advisers and 
other campus offices. In the same survey, students preferred text messaging to phone calls, emails, 
and social media (in that order).*? Perhaps having faculty share information with students could be 
more effective. 


Colleges participating in this study asked students to complete a survey on challenges they faced 
and their academic progress. However, very few students completed these self-reports. When asked 
about them during interviews, very few, if any, students remembered being asked to complete these 
surveys. But advisers found this information useful when it was available. It may be possible to take 
a similar approach to the informational messages to prompt action for students to complete these 
surveys. To boost survey completion rates, it also may be worthwhile to give students incentives, 
either financial or through priority registration. 


Colleges could also consider student focus groups and surveys to further understand what would 
be most salient and effective. 


e¢ When using predictive analytics, there can be a tension between complexity and utility. More 
transparency may lead to more effective practice. 


Advisers at MCCC disagreed with the predictive analytics risk assessment scores provided by the 
predictive analytics tool, whose algorithm includes many variables. However, because the algorithm 
is proprietary, the variables included are not public, and it is unclear which variables drive an in- 
dividual student’s score. Additionally, at UNC Charlotte, there was an error with the predictive 
analytics tool. This lack of clarity — and accuracy, in the case of UNC Charlotte — made it chal- 
lenging for advisers to know why a student had been identified as “at risk.” In order to account for 
this, during student’s second semester, risk was identified using course-specific grades, GPA, and 


38 Headlam, Anzelone, and Weiss (2018). 
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information gleaned from interactions with students, ignoring the risk score. This provided advisers 
with a measure of risk that was easier to address. 


This challenge raises questions about the utility of predictive analytics in student support. How 
useful is it to identify a student determined to be at risk if the adviser is unsure why the student is 
at risk? If an adviser does not know what is driving a student’s risk score, how they can intervene 
effectively? To maximize accuracy, these algorithms are complex and incorporate many variables. 
While this may improve predictions about a student’s likelihood of achieving positive outcomes, the 
increase in complexity may make the tool less helpful to advisers. Perhaps a simpler algorithm would 
be more useful, or more transparency is needed in terms of the variables included the algorithm or 
the variables that appear to be driving an individual student’s score. 


¢ Technology implementation may require more time and resources to lead to changes in student 
experiences. 


Advisers’ use of technology is central to the iPASS theory of change, as shown in Figure 1. Theoretically, 
the key program activities — from communication to the identification of struggling students to 
the material covered during advising appointment discussions — will be aided by technology, which 
should provide advisers with more data on students and increase efficiency.** This study’s early alert 
data shows that advisers at two of the three colleges were provided with more data on program group 
students. (These students received more early alerts than control group students.) However, at all 
the colleges, advisers described struggling to find time to become acclimated to the new technology 
tools, then use them. Advisers also struggled to find time to meet with all their assigned program 
group students. Though the goal of iPASS is to employ technology to make advisers’ lives easier, 
technology, to be effectively implemented, may require more time and resources from advisers than 
they have available. Furthermore, it may be important to explore providing additional resources to 
decrease adviser caseloads, allowing them more time to devote to individual students. 


Changes in infrastructure may be prioritized over changes in students’ experiences when incorporat- 
ing technology tools into student support and using it to spur institutional change. A key goal of this 
work is to streamline processes for advisers, which does provide clear benefits. It may allow them to 
use their time more efficiently and better identify and intervene with the most struggling students. 


That said, there is no guarantee that this streamlining will trickle down to changes in students’ 
experiences. Technology may provide better planning and information but not necessarily more 
support and guidance for students. Previous iPASS publications have emphasized the importance of 
not just deploying technology tools but providing support, guidance, and professional development 
for staff to ensure that technology is actually used and used effectively.” It is not yet clear how ef- 
fective technology use by faculty and advising staff members best translates into positive outcomes 
for students. Future research could further explore how advising interactions could make effective 
use of technology tools. 
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¢ In addition to improving student support, advising redesign work may present new opportuni- 
ties for growth and learning across institutions. 


Across the three schools, leadership roles for individuals emerged, as did new supporting roles, such 
as the peer mentors at Fresno. Institutions planning or implementing advising redesign may find it 
helpful to take stock of existing roles and leadership responsibilities — formal and informal — and 
consider how individual roles and responsibilities may evolve as advising redesign progresses. In 
addition to faculty, schools may consider including stakeholder groups such as students and execu- 
tive leadership, so they may contribute to moving the work forward, holistically and intentionally. 
Finally, institutions may want to consider ways to document advising redesign work (and rework) 
to identify challenges and opportunities that may arise. 


Conclusion 


Technology has the potential to support educational reform, providing the foundation for college 
advisers and staff to change how they interact with and support students. iPASS is an initiative 
designed to help colleges invest in technology and use it to redesign advising and student support. 
This report focused on three institutions’ efforts to enhance their existing iPASS work by increas- 
ing informational communication with students, improving the process through which struggling 
students are identifying and supported, and redesigning the advising session. 


The findings from the project show that the colleges were largely able to implement their enhance- 
ments, although overall they did not lead to a large difference in student experiences and did not 
have positive effects on their academic outcomes. 


Mounting evidence from the study of education reforms suggests that impacting students’ academic 
outcomes requires more substantial changes in their experiences. Reflecting a tension between scale 
and intensity, the study colleges managed to expand the iPASS enhancements to a broader group 
of students, but staff rarely had the time or resources to implement them fully, particularly finding 
enough time to devote to individual students. 


Adopting new technology and using it to redesign advising is an iterative process that takes time. 


Thus, this effort should be viewed as one step in the process to broader change. As such, it is hoped 
that the findings from the project can be a useful guide to colleges as they move forward. 
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APPENDIX 


APPENDIX TABLE A.1 Demographics of Study Sample, 
by Institution and Research Group 


OUTCOME 


University of California, Fresno 
Female (%) 


Race/ethnicity (%) 
Hispanic 
White 
Black 
Other 


Age (%) 
18 and under 
19-24 
25 and over 


College credits earned at study enrollment 


Sample size 


Montgomery County Community College 
Female (%) 


Race/ethnicity (%) 
Hispanic 
White 
Black 
Other 


Age (%) 
18 and under 
19-24 
25 and over 


College credits earned at study enrollment 


Sample size 


PROGRAM 


61.4 


CONTROL 


DIFFERENCE 


-0.6 


1.4 
0.5 
0.8 
-2.7 


0.9 
-0.9 
0.0 


0.14 


1.07 


STANDARD 
ERROR 


2.8 


2.8 
1.9 
0.9 
2.4 


1.8 
1.8 
0.0 


0.70 


1.8 


1.0 
1.8 
1.4 
1.4 


0.8 
1.8 
1.8 


0.99 


P-VALUE 


0.826 


0.623 
0.773 
0.394 
0.259 


0.613 
0.613 


0.871 


0.468 


0.790 
0.677 
0.702 
0.980 


0.775 
0.251 
0.200 


0.276 
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APPENDIX TABLE A.1 (continued) 


STANDARD 
OUTCOME PROGRAM CONTROL DIFFERENCE ERROR P-VALUE 
University of North Carolina, Charlotte 
Female (%) 43.7 43.1 0.7 1.6 0.680 
Race/ethnicity (%) 
Hispanic 9.1 10.0 -0.8 1.0 0.382 
White 59.9 60.3 -0.4 1.6 0.795 
Black 14.5 14.4 0.2 14 0.895 
Other 16.5 15.4 11 1.2 0.356 
Age (%) 
18 and under 29.5 28.4 1.2 1.5 0.436 
19-24 66.2 66.8 -0.5 1,5 0.722 
25 and over 4.2 4.8 -0.6 0.7 0.374 
College credits earned at study enrollment 29.62 29.80 -0.19 0.57 0.744 
Sample size 1,902 1,901 


SOURCE: MDRC calculations using data from California State University, Fresno, Montgomery County Community College, and 
University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true 
effect. A lower p-value suggests a higher likelihood of a nonzero effect. 

Estimates are weighted to account for differences in random assignment ratios across advisers. 

Spring and summer records are combined. Similarly, winter and fall records are combined. 
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APPENDIX TABLE A.2 Differences in Early Alerts and Advising Appointments 


STANDARD 
INSTITUTIONS PROGRAM CONTROL DIFFERENCE ERROR P-VALUE 
California State University, Fresno 
Had any early alerts (%) 
1st semester (with enhancements) 34.9 5.8 29.1 2.2 0.000 
2nd semester (with enhancements) 39.9 11.5 28.4 2.4 0.000 
3rd semester (after enhancements) 5.8 6.0 -0.2 1.4 0.909 
4th semester (after enhancements) 3.0 2.6 0.4 0.9 0.704 
Number of alerts 
1st semester (with enhancements) 0.59 0.08 0.50 0.04 0.000 
2nd semester (with enhancements) 0.75 0.15 0.60 0.05 0.000 
3rd semester (after enhancements) 0.08 0.09 -0.01 0.02 0.603 
4th semester (after enhancements) 0.04 0.03 0.01 0.01 0.661 
First year (with enhancements) 1.34 0.24 1.10 0.07 0.000 
Had any meetings with an adviser (%) 
1st semester (with enhancements) 96.9 44.1 52.8 2.1 0.000 
2nd semester (with enhancements) 92.5 54.5 38.1 2.2 0.000 
3rd semester (after enhancements) 48.0 49.5 -1.5 2.9 0.607 
4th semester (after enhancements) 52.6 50.6 2.0 2.9 0.477 
Number of meetings 
1st semester (with enhancements) 1.94 0.73 1.21 0.07 0.000 
2nd semester (with enhancements) 1.63 0.98 0.65 0.08 0.000 
3rd semester (after enhancements) 0.79 0.74 0.05 0.06 0.438 
4th semester (after enhancements) 0.82 0.80 0.02 0.06 0.681 
First year (with enhancements) 3.57 1.71 1.86 0.12 0.000 
Sample size 610 609 
Montgomery County Community College 
Had any early alerts (%) 
1st semester (with enhancements) 87.6 46.3 41.3 1.6 0.000 
2nd semester (with enhancements) 47.9 34.0 13.9 1.9 0.000 
3rd semester (after enhancements) 25.1 27.6 -2.5 1.7 0.149 
4th semester (after enhancements) 12.9 4.2 8.7 11 0.000 
Number of alerts 
1st semester (with enhancements) 2.45 0.88 1.57 0.07 0.000 
2nd semester (with enhancements) 1.47 0.68 0.79 0.07 0.000 
3rd semester (after enhancements) 0.55 0.59 -0.04 0.05 0.460 
4th semester (after enhancements) 0.37 0.06 0.31 0.04 0.000 
First year (with enhancements) 3.92 1.56 2.36 0.10 0.000 
Had any meetings with an adviser (%) 
1st semester (with enhancements) 79.2 57.0 22.2 2.0 0.000 
2nd semester (with enhancements) 54.4 49.2 5.2 2.2 0.021 
3rd semester (after enhancements) 47.6 53.0 -5.4 2.2 0.016 
4th semester (after enhancements) 28.5 33.9 -5.4 2.1 0.009 
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APPENDIX TABLE A.2 (continued) 


STANDARD 
INSTITUTIONS PROGRAM CONTROL DIFFERENCE ERROR P-VALUE 
Number of meetings 
1st semester (with enhancements) 1.32 0.90 0.41 0.05 0.000 
2nd semester (with enhancements) 1.39 1.40 -0.01 0.13 0.936 
3rd semester (after enhancements) 2.08 2.11 -0.03 0.19 0.877 
4th semester (after enhancements) 1.00 1.21 -0.21 0.14 0.130 
First year (with enhancements) 2.71 2.31 0.40 0.14 0.005 
Sample size 926 1,086 
University of North Carolina - Charlotte 
Had any early alerts (%) 
1st semester (with enhancements) 30.3 31.5 -1.2 1.5 0.432 
2nd semester (with enhancements) 17.5 17.3 0.3 12 0.824 
3rd semester (after enhancements) 14.3 15.1 -0.7 11 0.525 
4th semester (after enhancements) 9.8 10.6 -0.8 1.0 0.409 
Number of alerts 
1st semester (with enhancements) 0.45 0.48 -0.03 0.03 0.267 
2nd semester (with enhancements) 0.22 0.21 0.01 0.02 0.663 
3rd semester (after enhancements) 0.18 0.20 -0.02 0.02 0.306 
4th semester (after enhancements) 0.13 0.13 -0.01 0.01 0.491 
First year (with enhancements) 0.67 0.69 -0.02 0.03 0.506 
Had any meetings with an adviser (%) 
1st semester (with enhancements) 83.6 79.2 4.5 1.2 0.000 
2nd semester (with enhancements) 70.6 67.3 3.3 1.5 0.025 
3rd semester (after enhancements) 60.5 60.7 -0.1 1.6 0.937 
4th semester (after enhancements) 50.5 49.7 0.8 1.6 0.626 
Number of meetings 
1st semester (with enhancements) 1.37 1.29 0.07 0.04 0.040 
2nd semester (with enhancements) 1.19 1.12 0.07 0.04 0.065 
3rd semester (after enhancements) 1.11 1.13 -0.02 0.05 0.759 
4th semester (after enhancements) 0.88 0.86 0.02 0.05 0.724 
First year (with enhancements) 2.56 2.41 0.15 0.06 0.013 


Sample size 1,902 1,901 


SOURCE: MDRC calculations using data from California State University, Fresno, Montgomery County Community College, and 
University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true 
effect. A lower p-value suggests a higher likelinood of a nonzero effect. 

Estimates are adjusted by pre-random assignment characteristics of sample members. 

Estimates are weighted to account for differences in random assignment ratios across advisers. 

Spring and summer records are combined. Similarly, winter and fall records are combined. 

Meetings with advisors were mostly conducted in-person, but some were over the phone or via email. The data did not allow for 
accurate differentiation of the types of meetings. 
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APPENDIX TABLE A.3 Differences in Enrollment and Credits Attempted/Earned, 
by Timing of Registration Hold at Montgomery County Community College 


STANDARD 
OUTCOME PROGRAM CONTROL DIFFERENCE ERROR P-VALUE 
Enrollment in classes before registration hold (%) 
1st semester (with enhancements) 96.1 96.9 -0.9 0.5 0.092 
2nd semester (with enhancements) 50.4 53.4 -2.9 1.8 0.105 
3rd semester (after enhancements) 36.5 41.5 -5.0 1.8 0.005 
4th semester (after enhancements) 27.0 28.1 “11 1.6 0.492 
Enrollment in classes after registration hold (%) 
1st semester (with enhancements) 28.3 29.5 -1.2 1.7 0.455 
2nd semester (with enhancements) 10.4 13.0 -2.7 1.2 0.024 
3rd semester (after enhancements) 10.3 10.8 -0.5 1 0.677 
4th semester (after enhancements) 6.0 6.0 0.0 0.9 0.980 
Credits attempted before registration hold 
1st semester (with enhancements) 5.96 6.06 -0.10 0.11 0.345 
2nd semester (with enhancements) 3.78 3.85 -0.07 0.16 0.679 
3rd semester (after enhancements) 2.82 3.10 -0.29 0.16 0.082 
4th semester (after enhancements) 1.78 1.89 -0.11 0.12 0.359 
All semesters 14.33 14.90 -0.57 0.39 0.147 
Credits attempted after registration hold 
1st semester (with enhancements) 1.33 1.50 -0.17 0.10 0.090 
2nd semester (with enhancements) 0.41 0.50 -0.08 0.05 0.097 
3rd semester (after enhancements) 0.46 0.43 0.02 0.05 0.660 
4th semester (after enhancements) 0.23 0.25 -0.02 0.04 0.606 
All semesters 2.43 2.68 -0.25 0.15 0.097 
Credits earned before registration hold 
1st semester (with enhancements) 4.25 4.42 -0.17 0.12 0.144 
2nd semester (with enhancements) 2.94 3.05 -0.11 0.15 0.470 
3rd semester (after enhancements) 2.29 2.53 -0.24 0.15 0.108 
4th semester (after enhancements) 1.34 1.55 -0.20 0.11 0.062 
All semesters 10.64 11.31 -0.67 0.37 0.071 
Credits earned after registration hold 
1st semester (with enhancements) 1.07 1.20 -0.13 0.09 0.147 
2nd semester (with enhancements) 0.33 0.36 -0.04 0.04 0.414 
3rd semester (after enhancements) 0.33 0.33 -0.01 0.05 0.866 
4th semester (after enhancements) 0.18 0.18 0.00 0.03 0.919 


All semesters 1.90 2.06 -0.16 0.13 0.219 


Sample size 1,248 1,741 


SOURCE: MDRC calculations using transcript data from Montgomery County Community College. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true effect. A 
lower p-value suggests a higher likelihood of a nonzero effect. 

Estimates are adjusted by pre-random assignment characteristics of sample members. 

Estimates are weighted to account for differences in random assignment ratios across advisers. 

Spring and summer records are combined. Similarly, winter and fall records are combined. 

Transcript records for summer 2019 are not included in the fourth semester estimates. Few students enrolled in any classes during the summer 
terms. 
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APPENDIX TABLE A.4 Differences in Course Outcomes and Degrees Earned 


STANDARD 
INSTITUTIONS PROGRAM CONTROL DIFFERENCE ERROR P-VALUE 
California State University, Fresno 
Received a D or F in any course (%) 
1st semester (with enhancements) 52.1 49.2 2.9 2.8 0.305 
2nd semester (with enhancements) 47.3 48.9 -1.6 2.9 0.572 
3rd semester (after enhancements) 41.4 39.1 2.3 2.8 0.417 
4th semester (after enhancements) 34.8 33.3 1.5 27 0.572 
Withdrew from any course (%) 
1st semester (with enhancements) 1.5 14 0.4 0.7 0.544 
2nd semester (with enhancements) 2.6 1.5 1.2 0.8 0.153 
3rd semester (after enhancements) 1.5 1.0 0.5 0.6 0.410 
4th semester (after enhancements) 14 1.5 -0.4 0.7 0.576 
Earned Bachelor's degree (%) 0.6 0.2 0.5 0.4 0.207 
Sample size 610 609 
Montgomery County Community College 
Received a D or F in any course (%) 
1st semester (with enhancements) 32.8 33.6 -0.8 17 0.645 
2nd semester (with enhancements) 15.1 15.8 -0.6 1.3 0.625 
3rd semester (after enhancements) WA 10.5 0.6 14 0.573 
4th semester (after enhancements) 8.0 7.3 0.7 1.0 0.494 
Withdrew from any course (%) 
1st semester (with enhancements) 171 15.7 1.4 1.4 0.300 
2nd semester (with enhancements) 9.2 10.8 -1.6 1 0.153 
3rd semester (after enhancements) 6.3 8.4 -2.2 1.0 0.023 
4th semester (after enhancements) 4.3 4.3 0.0 0.7 0.992 
Earned any degree or certificate (%) 20.4 21.4 -1.1 1.5 0.473 
Sample size 1,248 1,741 
University of North Carolina, Charlotte 
Received a D or F in any course (%) 
1st semester (with enhancements) 32.4 32.4 0.1 1.5 0.971 
2nd semester (with enhancements) 29.6 28.5 1.2 1.5 0.428 
3rd semester (after enhancements) 26.1 25.6 0.5 1.4 0.722 
4th semester (after enhancements) 18.4 18.7 -0.3 1.3 0.784 
Withdrew from any course (%) 
1st semester (with enhancements) 23.8 24.1 -0.3 1.4 0.836 
2nd semester (with enhancements) 21.4 19.9 1.5 1.3 0.260 
3rd semester (after enhancements) 17.8 17.8 0.0 le 0.982 
4th semester (after enhancements) 12.9 13.9 -1.0 Al 0.371 
Earned Bachelor's degree (%) 47 5.0 -0.3 0.7 0.623 
Sample size 1,902 1,901 
(continued) 
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APPENDIX TABLE A.4 (continued) 


SOURCE: MDRC calculations using transcript data from California State University, Fresno; Montgomery County Community College; 
and University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

The p-value indicates the likelihood that the estimated impact (or larger) would have been generated by an intervention with zero true 
effect. A lower p-value suggests a higher likelihood of a nonzero effect. 

Estimates are adjusted by pre-random assignment characteristics of sample members. 

Estimates are weighted to account for differences in random assignment ratios across advisers. 

Spring and summer records are combined. Similarly, winter and fall records are combined. 

Transcript records for summer 2019 are not included in the fourth semester estimates. Few students enrolled in any classes during 
the summer terms. 
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APPENDIX FIGURE A.1 Timeline of iPASS Initiative 


Cohort 1 receives program 


Cohort 2 receives program 


Ongoing research activities 


19 colleges 26 colleges RCT colleges 


are selected 


receive iPASS | receive iPASS II 
grant grant 
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